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ABSTRACT 


This  work  is  the  result  of  a  study  of  G.  B.  Dantzig's  [1]  algorithm 
for  nonlinear  programs  where  the  objective  as  well  as  inequality  constraints 
are  convex  functions.  This  algorithm  consists  of  finding  an  equivalent 
"generalized  linear  program"  where  the  standard  procedure  for  determining 
the  candidate  to  enter  the  basis  is  replaced  by  an  auxiliary  nonlinear  pro¬ 
gram  where  the  nonlinear  constraints  of  the  original  problem  have  disappeared. 
The  algorithm  described  in  this  paper  is  very  close  to  Dantzig's  algorithm, 
but  the  relations  between  the  two  algorithms  are  not  yet  well  established. 


CONVEX  PROGRAMMING  -  DUAL  ALGORITHM 


I.  Definitions -Proper  ties 


The  initial  program 


PROBLEM  1: 


Find  x  such  that  0  (x)  is  maximum 

ro 

subject  to  0  (x)  0  i  =  1,2,..., m  . 


where  0q  and  0^  are  continuously  differentiable  concave  functions  in 

We  will  call.  0(x)  or  0  the  vector  whose  elements  are  0^(x)  . 

We  assume  that  the  optimal  value  of  0  (x)  is  finite. 

o 

Let  r.  •■=  (tt.  ,7T_, . . .  ,7T  )  be  a  row  vector. 

12  m 

Consider  the  following  program- 


PROBLEM  3: 


Find  x  such  that  ^/(x,7r)  is  maximumt 

where  =  t0(x)  +  0q(x)  and  7T  is  given  (tt  ;>  0)  . 


lat  s(tt)  be  an  optimal  solution  of  Problem  3. 


Assumption  1:  z(-tt)  is  continuous  in  7 r  and  finite  for  all  ir  0  . 
Let 


(1)  Q(r)  =  7T0(z )  +  0Q(Z)  - 

THEOREM  1:  The  gradient  of  ©(77)  in  the  7T-epace  is  equal  to  0(z)  . 
PROOF .  At  the  optimum,  z(tx)  ,  of  Problem  3  we  have 


(2) 


d^/(x, n)  =  0  (Maximum  with  respect  to  x) 


(3) 


jglZL 1  =  ^(x^)  M  +  frK***)  =  rt(z)  . 

d7T  bx  dr:  an  r  v 


THEOREM  2:  9(tt)  =  ^(x, tt)  '  is  a  convex  function  in  7T  . 

PROOF:  Let  V  and  V  be  two  scalars  such  that  V  +  X"  =  1  ,  and  ~h'  ,  V_>  0. 
Let  tt'  ,  7T"  ;>  0  be  two  values  of  7r  ;  and  z'  ,  z"  the  corresponding 
optimal  values  for  Problem  3*  Then, 

(4)  0"  -  9’  =  tt"0"  +  0^  *  (tt'0  +  0^) 

^7T"0'  +  0;  -  (tt'0'  +  0;) 

because  9(7t")  =  Max  (^(x,7T")} 

=  (tt"  -  tt’)  0' 

or  9"  -  9'  ^  (tt"  -  tt')  by  Theorem  1 

7T=7T' 


Figure  1 

i.e.,  the  variation  of  9(7:)  is  always  greater  than  or  equal  to  that  tangent 
plane,  which  characterizes  a  convex  function. 


-2- 


Let  the  following  program  be: 


PROBLEM  2:  [Find  77  ;>  0  such  that  9(77)  is  minimum 

Assumption  2:  There  exists  a  finite  optimal  solution  ft  to  Problem  2. 

THEOREM  3:  The  optimal  value  of  the  objective  function  of  Problem  2  is 
equal  to  0Q(ft)  the  optimal  value  of  the  objective  function 
of  Problem  1. 

PROOF:  A  necessary  condltior.  of  optimality  for  Problem  2  is 

AQ 

(5)  71  dir  =  0  (Kuhn-Tucker  condition) 

It  follows  that 

(6)  ft0(ft)  =  0  , 

and  — 

(7)  ®(ft)  =  0o(x)  . 

II.  The  Algorithm 

We  replace  the  initial  Problem  1  by  Problem  2  of  which  the  optima_ 
solution  yields  the  optimal  solution  for  Problem  1. 

Problem  2  is  the  minimization  of  a  convex  function,  9(77)  ,  where  the 
variable  7 7  is  subjected  to  being  nonnegative.  The  expression  of  the  func 
tion  9(7:)  is  not  given,  but  one  can  compute  its  value  for  all  77  ,  using 
the  auxiliary  program  Problem  3*  This  auxiliary  program  gives  us  also  the 
value,  0(z)  of  the  gradient  of  9 (77)  at  any  given  77  . 

We  can  solve  Problem  2  as  follows: 
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It 

(X1  Start  with  r,  =  if  ^  0  . 

k  k 

Compute  0(  7T )  and  0[z(7T  )]  using  Problem  3. 
Determine  the  vector  g(ir  )  defined  by 


(8) 


=  0i[z(tk)]  if  0.^  £  0  and/ or  if  7T\  >  o  . 
=  0  otherwise  (i.e.,  if  0^  >  0  and  7r  = 


0)  . 


In  order  to  simplify  the  notation  g  will  be  considered  in  some  cases  to  be 
a  rcw  vector  and  in  some  other  cases  to  be  a  column  vector. 


(9) 


k 

©  On  the  half -line  tt  =  ir“  +  "K  (t*  )  >  >  0  ,  determine 

s 

a  new  point  tt  '  £  0  at  a  finite  distance  corresponding 

to  ^  -  7^  £  0  and  satisfying  the  condition 


/ 


k  ! 


0  <  ^  t  Min 


l 

g,  <  0  l 

-  I 


0  Compute  Q(7T’  )  using  Froblem  3 


k  k  k 

If  9 {it  1  )  <  Q(7T  )  ,  go  to  1  ,  replacing  it  by 

k+1  k 
tt  =  it'  . 


If  0(?r'k)  0(77k)  and  if  rk  is  not  the  optimal 

solution,  ve  have  necessarily 


(10) 


(n,k  -  7Tk)  •  0(7Tk)  <  0 

(7T,k  -  7T,k)  •  0('n"k)  >  0  . 
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Figure  2 


Determine  then  a  new  point 


(11) 


rr"k  =  7rk  +  (l/2)?^g(7rk) 


)t  ^ 

i.e.,  the  middle  point  of  (7T  ,  tt'  )  or  smother  intermediate 

k  k 

point,  taking  into  account  the  respective  slopes  at  7T  and  7T'  . 

Jc 

Then  repeat  Q)  ,  with  it"  instead  of  hr  *  ,  and  so  on  until 

k+1 

we  obtain  (in  a  finite  number  of  steps)  a  point  tt  such 
that 


(12) 


e(7Tk+1)  <  e(7rk)  . 


REMARK:  To  prove  the  convergence,  we  have  to  perform  the  subdivision  of 


once  more,  which  yields  a  point  if 


tk+1 


,  and  work  with  the  better  of 


7:*k+1  and  7Tk"^"t'»  In  order  to  simplify  the  notations,  we  will  call 

k+1 

the  selected  point  7T  ,  and  7^  the  value  of  7\  for  that  point. 


k+1  k 

(§)  Go  to  @  with  7 r  instead  of  tt 
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III.  Convergence 


We  can  show  easily  (see  Q)  of  the  algorithm),  that  the  variation  of 
Q(tt)  ,  at  each  step  k  ,  is  strictly  negative,  except  if  g(?r  )  =  0  which 
implies  that  0[z(7f  )]  ^  0  and  ir  *0[z(tt  )]  =  0  .  In  these  conditions,  we 
achieved  the  optimum  of  Problem  2. 

Since  the  minimum  of  0(70  ,  tt  ^  0  is  equal  to  0q(&)  >  optimum 
of  Problem  1  which  is  assumed  to  be  finite,  the  sequence  of  points  tt  gives 
a  sequence  of  convergent  values  0(7;  )  because  we  have  a  lower  bound  for 
these  values.  Moreover,  we  will  make  the  following  assumption: 

Assumption  The  second  partial  derivatives  of  0(7t)  are  bounded  above  in 
all  directions  by  a  scalar  D  >  0  . 

One  can  show  that: 

2 

(13)  0(7Tk)  -  0(7Tk+1)  ^  (1/2)  *Jgk|  if  1/D  , 

I  k|2 

(14)  0(7Tk)  -  0(7Tk+1)  £  1/4  1^1  if  \  1/D 

II  k  k 

g|  =  modulus  of  g  ,  letting  g  =  g(7T  )  . 

Since  is  either  determined  by  (g)  or  equal  to  a  fraction  of  the  first 

value  that  we  tried  at  the  beginning  of  each  step  which  can  be  taken 

equal  to  a  constant  h  ,  for  all  k  . 

We  obtain  for  all  such  that  0  £  £  h  . 

(15)  e(7rk)  -  o(7rk+1)  ;>  k.\  !gk|2  , 

with  K  =  Min  (l/2  ,  1/lthD) 


from  which  we  deduce  that  the  series 


converges  and  that 
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1  t- 

(16)  ask-*« 

For  each  step  k  ,  by  the  convexity  of  9(7r)  we  have 

(17)  0  £  e(7Tk)  -  ©(tt)  £  (7Tk  -  7r)  •  0k 
where  7t  is  an  optimal  solution  of  Problem  2. 

By  definition  of  g  : 

(18)  (7Tk  -  ft)  0k  =  -  (7Tk  -  ft)  gk  +  (7Tk  -  ftN)  •  0k 

where  0^  is  a  vector  containing  only  "the  components  of  0  such  that 
0^  >  0  ,  71^=  0  .  Then 

(19)  (7Tk  -  ft  )  *  0k  =  -ft  •  0k  <  0  , 

\±.y)  '7rN  7V  rN  1 N  *  ' 

and  finally 

(20)  0  £  Q(7Tk)  -  ©(ft)  £  -(7Tk  -  ft)  gk  , 

and  in  particular 

(21)  (7Tk  -  ft)  \  .  gk  ^  0 

^.1 

which  means  that  the  vector  (tt  -  7T  )  has  an  acute  angle  with 
(ft  -  7Tk) 

It  follows 

(22)  I  ft  -  7Tk+1  |  2  £  |  ft  -  ^  I  2  +  I  7Tk+1  -  7Tk  |  2  = 

|ft  -  7T  |  +  |  g  | 

Summing  with  respect  to  k  : 

k 

(23)  |  7T  ~  7T  |  ^  |  7T  -  7T  |  +  ^  \'  I6  I 

k'=0 
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\  k'  2 

Since  >  X  ,  |  g  J  converges  and  7^,  £  h  ,  the  serie? 


Xv 


k'  .2 


converges  when  k  -*  »  }  from  which  we  obtain  an  upper  bound  for  the  distance 


it  -  'iff  |  . 


We  have  to  consider  two  cases : 


CASE  1:  g  -*  0  ,  at  least  for  a  finite  or  infinite  subsequence  of  k  .  Then 
the  corresponding  subsequence 

(24)  gk  (TTk  -  ft)  ■*  0  and  0(iTk  -*  ©(ft) 

For  these  conditions,  the  algorithm  converges  to  an  optimal  solution. 

CASE  2 :  g  does  not  converge  to  zero  for  any  subsequence  of  k  .  This  means 

1c  i  k  i 

g  is  different  from  zero  for  all  k  ,  and  moreover  |  g  |  has  a 

lower  bound  >  0  .  For  these  conditions,  the  convergence  of  the 


implies  the  convergence  of  the  series 


\  *  U  P 

series  )  7^.  |  g  |  implies  the  convergence  of  the  series 

X  \  I gk  I  X  7^.  and  therefore  implies  the  convergence  of 


the  point  tt  which  was  defined  by 


rk  -  X  v gk’ 


k  k 

Let  ft  be  the  limit  of  tt  ,  and  let  g  be  the  limiting  value  of  g  at  that 

point.  (This  limit  exists  by  Assumption  1.  ) 

\  k 

The  convergence  of  the  series  )  7  implies  that  7^  -*•  0  and  by 
Assumption  5,  7^  is  determined  by  the  relation  (9)  f°r  K  sufficiently 

large.  There  exists  at  least  one  index  i  and  two  subsequences  K'  and  K" 
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corresponding  to  the  steps  k  such  that. 


(26)  k'  £  K" 


k  1 

<  o 


7T*>  0 


IT, 


V  " 


0i'  >o 


and 


(27)  k"  £  K" 


k" 

g*  >  o 


k" 

7T*  =  0 


k" 

0i  <  0 


k  *  k" 

So  0i  and  0^  go  to  zero,  and  we  have  by  the  fact  that  the  second  order 


derivatives  of  0  have  an  upper  bound,  D  , 


(28) 


k*  ^  _  k' 

-gj  £  ® 


If  j  is  the  index  of  a  nonzero  component  of  §  ,  we  have  for  k'  sufficiently 
large. 


(29) 


Vlsk/l-- V'*k'  h5ls'ls5lpJ>° 


We  have  here  a  contradiction  with  the  fact  that  7T  converges  to  tt  which 

k 1  ~ 

implies  that  gj  -*  0  .  Thus  the  components  of  g  must  be  zero,  which 

contradicts  the  assumption  of  a  possible  second  case. 


IV.  REMARKS 

l)  Nature  of  the  Algorithm;  Our  initial  program  is 

Find  x  such  that  0  (x)  is  maximum 

ro 

subject  to  0i(x)  ^  0 
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Problem  1 


Let  us  consider  the  dual  [2]  of  Problem  1: 


(D) 


Find  7T  ^  0  such  that  f(x,7r)  is  minimum 

subject  to  ^(x>7t)  =  0  j  =  1,2,... ,n  . 

J 


We  see  that  solving  the  auxiliary  program  Problem  3  for  a  given  nonnegative  7T 
is  equivalent  to  finding  a  feasible  solution  (x,7r)  for  the  dual  program 
(D)  .  Problem  2,  which  minimizes  9(7t)  solves  (D)  by  a  discrete  gradient 


method. 


2)  On  Assumption  1;  Assumption  1  seems  at  first  very  restrictive, 
nevertheless,  if  the  optimum  of  Problem  1  is  finite,  one  can  always  remove 
the  constraints  ^(x)  and  the  functional  0q(x)  far  enough  from  the 
origin  in  order  to  have  z(7r)  at  a  finite  distance.  For  instance,  this 
modification  can  consist  in  Joining  the  surfaces  y  =  ^(x)  and  yQ  =  0q(x) 
to  the  vertical  cylinder  lx  j  =  constant  if  the  constant  is  chosen  large 
enough.  This  would  reduce  the  search  for  an  optimum  of  Problem  3  to  the 
interior  of  a  sphere,  (x)  =  constant.  One  could  imagine  other  compact  spaces. 

The  continuity  of  z(7r)  for  7T  can  be  achieved,  e.g.,  by  assuming  the 
optimum  of  Problem  3  is  always  unique.  This  is  the  case  if  ^(x,7r)  is  strictly 
concave  in  x  ,  i.e.,  if  one  at  least  of  the  functions  0^(x)  or  0Q(x)  is 


strictly  concave. 
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